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Outcome-driven  Service  Provider  Performance  under 
Conditions  of  Complexity  and  Uncertainty 


Presenter:  Kevin  S.  Buck  co-leads  research  of  web  services  performance  in  tactical  environments  and 
stakeholder-driven  performance  management.  He  provides  investment  and  portfolio  management 
support  to  Government  sponsors.  Buck  has  a  BS  in  Marine  Transportation  from  the  U.S.  Merchant 
Marine  Academy,  an  MS  in  Industrial  Administration,  and  an  MBA. 

Author:  Diane  P.  M.  Hanf  co-leads  research  of  web  services  performance  in  operational  environments. 
She  provides  modeling  and  analysis,  software  acquisition,  and  test  and  evaluation  support  to  Government 
sponsors.  Hanf  has  Bachelor’s  degrees  in  Electrical  Engineering,  Wire  Communications  Technology  and 
Business  Administration  and  an  MS  in  Systems  Engineering. 


Abstract 

While  Service-oriented  Architecture  (SOA)1  can  help  organizations  share  resources  and 
leverage  economies-of-scale,  it  can  increase  acquisition  complexity  (e.g.,  multiple  new/different 
relationships  to  manage)  and  uncertainty  (e.g.,  nature/magnitude  of  future  service  demands). 
Given  this  additional  complexity  and  uncertainty,  MITRE  developed  a  performance  management 
framework  to  help  Government  organizations  measurably: 

■  Articulate  SOA  outcomes  and  identify  outcome  drivers; 

■  Define  SOA  technical  and  acquisition  performance  metrics  through  the  application  of 
Return-on-lnvestment  (ROI)  principles  and  monitor  performance  as  a  comparison  of 
current  delivery  to  initial  ROI  expectations; 

■  Translate  SOA  objectives  into  contractor  performance  management  mechanisms. 

This  paper  describes  applying  ROI  analysis  principles  for  SOA  performance 
management,  creating  Service-level  Agreements  (SLAs)  to  articulate  agreements  between  the 
Government  and  external  service  providers,  and  managing  SLAs  through  a  governance 
framework  (Hanf  &  Buck,  2009,  March). 

This  white  paper  highlights  key  findings  of  research  undertaken  by  The  MITRE 
Corporation  (MITRE)  and  the  resulting  recommendations  for  (1)  applying  Return-on-lnvestment 
(ROI)  analysis  principles  as  the  foundation  for  more  effective  performance  management  of 
Government  Service-oriented  Architecture  (SOA),  (2)  creating  comprehensive  Service-level 
Agreements  (SLAs)  to  articulate  agreements  between  the  Government  and  external  service 
providers,  and  (3)  managing  SLAs  through  a  governance  framework  (Oakley-Bogdewic  &  Buck, 
2009;  Hanf  &  Buck,  2009,  March  25).  As  illustrated  in  Figure  1,  MITRE’s  recommendations 
address  the  additional  managerial  complexity  and  uncertainty  that  SOA  objectives  and 
proposed  solutions  often  create. 


1  SOA  is  an  architectural  style  that  guides  all  aspects  of  creating/applying  business  processes  through 
service  packaging  and  defines/provisions  the  Information  Technology  (IT)  infrastructure  (Newcomer  & 
Lomow,  2005). 
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To  Address 
V-  Complexity  and 
Uncertainty 


J 


Figure  1.  Key  Aspects  of  MITRE’s  Research 


1.0  The  Importance  of  Performance  Management 

Mechanisms  are  currently  not  consistently  in  place  within  the  Federal  Government  for 
programs  to  identify  key  stakeholders,  quantitatively  articulate  stakeholder  needs,  and 
quantifiably  assess,  on  a  timely  basis,  whether  stakeholder  needs  have  been  satisfied.  The 
new  Administration  is  currently  focused  on  a  key  symptom  of  such  improperly  functioning 
mechanisms:  lack  of  transparent  accountability.  President  Barack  Obama  explained: 

My  Administration  is  committed  to  creating  an  unprecedented 
level  of  openness  in  Government.  We  will  work  together  to  ensure  the 
public  trust  and  establish  a  system  of  transparency,  public  participation, 
and  collaboration.  Openness  will  strengthen  our  democracy  and 
promote  efficiency  and  effectiveness  in  Government.  Government 
should  be  transparent.  Transparency  promotes  accountability  and 
provides  information  for  citizens  about  what  their  Government  is  doing. 
Information  maintained  by  the  Federal  Government  is  a  national 
asset.  My  Administration  will  take  appropriate  action,  consistent  with  law 
and  policy,  to  disclose  information  rapidly  in  forms  that  the  public  can 
readily  find  and  use.  Executive  departments  and  agencies  should  harness  new  technologies  to 
put  information  about  their  operations  and  decisions  online  and  readily  available  to  the 
public.  Executive  departments  and  agencies  should  also  solicit  public  feedback  to  identify 
information  of  greatest  use  to  the  public.  (2009) 

Over  the  past  year,  lackluster  demonstration  of  effective  Government  performance  has 
resulted  in  the  establishment  of  new  regulations/requirements  that  compel  agencies  to  more 
frequently  and  credibly  communicate  the  value  delivered  by  Government  programs  in  exchange 
for  funds  provided  by  stakeholders.  The  requirement  for  a  Performance  Improvement  Officer  is 
one  of  the  provisions  of  an  executive  order  signed  on  November  13,  2007,  to  compel  agencies 
to  derive  better  results  from  their  programs.  Agencies  will  now  be  required  to  demonstrate 
robust  performance  management  efforts,  including  the  development  or  improvement  of  strategic 
plans  and  aggressive  and  accurate  measurement  of  progress  in  achieving  overarching 
performance  goals. 

Current  reporting  requirements  for  programs  and  expenditures  will  likely  be  more  closely 
scrutinized  for  realism,  consistency,  accuracy,  and  alignment  with  strategic  objectives.  The 
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Comptroller  General  asks  federal  programs  and  agencies  to  improve  performance  management 
by: 


■  Comprehensively  reassessing  what  the  federal  government  does  and  how  it  does  it: 
reconsidering  whether  to  terminate  or  revise  outdated  programs  or  services 
provided. 

■  Reexamining  the  beneficiaries  of  federal  programs:  reconsidering  who  is  eligible  for, 
pays  for,  and/or  benefits  from  a  particular  program  to  maximize  federal  investments. 

■  Improving  economy,  efficiency,  and  effectiveness  of  federal  operations:  capturing 
opportunities  to  reduce  costs  through  restructuring  and  streamlining  federal  activities. 

■  Attacking  activities  at  risk  of  fraud,  waste,  abuse,  and  mismanagement:  focusing  on 
minimizing  risks  and  costs.  (Mihm,  2000,  July  20) 

Regardless  of  whether  government  programs  operate  in  Information  Technology  (IT)- 
intensive  environments  or  not,  there  is  increasing  momentum  toward  sharing  of  resources, 
solutions,  and  risks  across  government  organizations/Agencies.  While  this  trend  supports 
leveraging  synergy,  reducing  stovepipes/redundancy,  and  economies-of-scale,  it  often 
increases  acquisition  complexity  (e.g.,  multiple  new  and  different  relationships  to  manage)  and 
uncertainty  (e.g.,  nature  and  magnitude  of  future  demands  for  supplies/services  that  are 
currently  being  acquired).  Given  the  increased  emphasis  on  transparency  and  accountability  for 
Federal  government  expenditures  to  our  ultimate  stakeholders  (e.g.,  the  taxpayer)  in  an 
environment  with  increased  acquisition  complexity  and  uncertainty,  foundational  steps  that  we 
recommend  include: 

a)  Understanding  an  organization’s  own  performance  with  respect  to  its  stakeholders’ 
expectations, 

b)  Finding  ways  to  effectively  communicate  performance  in  the  right  form  and  at  the 
right  time  to  ultimate  stakeholders.  This  allows  for  expectations  to  be  effectively 
managed  and/or  course  corrections  to  be  accomplished  before  resources  are 
unnecessarily  expended  for  too  long  on  objectives  that  are  no  longer  worthwhile  or 
on  solutions  that  will  not  succeed,  and 

c)  Establishing  mechanisms  to  readily  re-calibrate  performance  needs/expectations  as 
the  future  becomes  less  uncertain. 

We  discuss  the  following  recommendations  in  the  context  of  an  SOA  environment: 

■  Application  of  an  ROI-based  performance  management  framework  to  support 
sponsors  in  aligning  operational  and  contract-related  performance  metrics  with 
monetizable  and  non-monetizable  costs,  benefits,  and  risks  deemed  critical  for 
achievement  of  desired  outcomes, 

■  A  performance  execution  process  based  on  SLAs  as  a  key  means  of  communicating 
and  monitoring  performance,  and 

■  An  SLA  governance  framework  that  enables  decision-makers  to  manage  SLAs  as  an 
on-going  re-evaluation  of  what  performance  matters  to  stakeholders. 
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2.0  SOA  Performance  Management 

As  Federal  Government  agencies  transform  their  enterprises  to  be  service-oriented,  a 
disciplined  process  to  effectively  and  efficiently  manage  both  operational 
and  service  provider  performance  has  yet  to  be  widely  embraced.  In  the 
absence  of  such  a  process,  program  and  portfolio  managers  are  often 
challenged  to  clearly  and  measurably  connect  SOA  stakeholder  needs, 
desired  outcomes,  and  operational,  technical,  and  service  provisioning 
performance.  As  is 
illustrated  in  Figure  2-1 , 

SOA  is  an  architectural 
approach  used  to  build 
solutions  that  contain  a 
set  of  services,  service 
consumers,  service  producers  and  service 
contracts  (Logan,  2008). 


SOAP  =  Simple  Object  Access  Protocol 

UDDI  =  Universal  Description ,  Discovery ;  and  Integration 

WSDL  =  Web  Services  Description  Language 


SLAsforSOA  Performance 
Management 


* 


SOA  SLA  Governance 


Figure  2-1.  The  SOA  Construct 

Through  Government  case  observations  and  investigation  of  methodologies  successfully 
employed  within  commercial  industry,  the  MITRE  research  team  developed  a  performance 
management  framework — discussed  later  and  shown  in  Figure  4-1 — which  guides  framework 
users  (e.g.,  multiple  SOA  participants  in  different  stages  of  the  SOA  lifecycle)  through  key 
decisions  that  will  need  to  be  made  in  effectively  and  efficiently  managing  performance  of  SOA 
implementations.  The  SOA  (Newcomer  &  Lomow,  2005)  performance  management  framework 
helps  Government  portfolio/program  managers  and  system/performance  engineers: 

■  Measurably  articulate  expected  SOA  outcomes  and  identify  outcome  drivers, 

■  Define  and  monitor  technical  and  acquisition  performance  metrics  through  the 
application  of  Return-on-lnvestment  (ROI)  principles  and  the  on-going  comparison  of 
current  delivery  to  initial  SOA  ROI  expectations,  and 

■  Effectively  translate  operational  and  overarching  government  SOA  performance 
objectives  into  contractor  performance  management  mechanisms. 
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MITRE’s  recommendations  target  critical  challenges  associated  with  the  increased 
complexity  and  uncertainty  that  are  often  created  by  SOA.  They  also  mitigate  the  risk  of 
measurement  overload  (i.e.,  “losing  the  forest  for  the  trees”)  by  providing  a  mechanism  to  derive 
vital  and  coherent  outcome-focused  metrics.  These  critical  challenges  are  summarized  in  the 
paragraphs  that  follow: 

Expectations  of  Savings  without  Analysis 

For  Government  agencies  and  programs  that  have  made  decisions  to  adopt  SOA,  robust 
and  repeatable  methods  for  effectively  and  efficiently  managing  performance  over  the  lifecycle 
have  not  emerged.  For  many  Federal  Government  organizations,  there  is  a  mindset  that, 
because  SOA  is  mandated,  rigorous  investment  analysis/management  is  not  necessarily  an 
urgent  requirement.  The  expectation  that  SOA  will  save  money  has  resulted  in  already 
decreasing  funding  profiles  for  programs,  increasing  the  criticality  of  developing  and  applying 
methodologies  that  result  in  selection  of  cost-effective  strategies  and  solutions.  These 
methodologies  should  directly  relate  to  fulfilling  stakeholder  needs,  closing  capability  gaps,  and 
achieving  multiple  outcomes. 

Understanding  the  SOA  Lifecycle 

One  challenge  in  effectively  managing  SOA  performance  is  the  lack  of  relevant  SOA 
lifecycle  performance  benchmarks  that  Government  programs  can  leverage  to  determine 
realistic  SOA  outcomes  and  performance  thresholds.  This  lack  increases  the  degree  of 
uncertainty  regarding  what  can  realistically  be  expected  from  SOA.  The  current  lack  of 
benchmarks  is  primarily  a  symptom  that  (a)  many  sponsors  are  still  in  the  initial  planning  or 
development  stages  with  SOA  and  do  not  have  on-going,  steady-state  results  to  share  yet,  and 
(b)  those  organizations  that  do  have  steady-state  performance  results  often  consider  the 
information  to  be  proprietary,  requiring  close-hold.  In  the  absence  of  meaningful  benchmarks 
from  referent  organizations,  alternative  methods  must  be  implemented  by  sponsors  to  evaluate 
performance  of  the  potentially  substantial  investments  in  SOA  that  will  be  undertaken  by 
numerous  participants  in  SOA  (e.g.,  SOA  developers,  service  producers,  and  service 
consumers). 

Measuring  Non-fiscal  Returns 

Our  research  confirms  that  SOA  expected  returns  are  not  always  fiscally  driven  (e.g., 
compliance  with  law  and  regulation  or  loss  of  life  is  more  important  in  many  cases),  and  the 
SOA  construct  seeks  to  align  mission  and  investments  that  involve  promoting  a  service-oriented 
culture.  As  a  consequence,  the  research  team  proposes  an  expanded  definition  of  ROI,  to 
include  return  on  closing  capability  gaps  that  are  targeted  by  an  SOA  implementation  that 
includes  non-monetizable  value  propositions  such  as  compliance  with  law  and  regulatory 
mechanisms,  avoidance  of  loss  of  life  and  customer  (e.g.,  government  user  or  citizen) 
satisfaction.  This  definition  is  illustrated  in  Figure  3-1. 
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(5)  Stoplight  matrix  of  risk  assessment 
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Figure  3-1.  ROI  Analysis  Considerations  for  SOA 


Expectation  Management 

Application  of  the  expanded  ROI  methods,  in  an  on-going  performance  management 
program,  involves  comparison  of  actual  tangible  and  intangible  results  realized  from  selected 
SOA  investments  to  realistic,  initial  investment  expectations.  Initial  expectations,  in  and  of 
themselves,  should  reflect  an  incremental  comparison  of  proposed  SOA  investment  returns  to 
those  anticipated  should  current  approaches  be  continued  (i.e.,  the  status  quo,  or  “do  nothing” 
case).  The  value  of  ROI  analysis  for  an  on-going  performance  management  program  must  be 
balanced  against  the  resources  required  to  perform  the  analysis  and  will  also  greatly  depend 
upon  the  ability  of  Government  sponsors  to  effectively  characterize  initial  expectations  from 
SOA  in  measurable  terms.  According  to  ZapThink  Research,  “only  by  understanding  the  full 
range  of  SOA  value  propositions  can  companies  begin  to  get  a  handle  on  calculating  the  ROI  of 
SOA”  (Schmelzer,  2005). 


Effective  SOA  Management  Can  Be  Resource-intensive 

Application  of  ROI  principles  for  SOA  performance  management  will  likely  increase 
resources  devoted  to  planning  and  monitoring  efforts.  ROI  analysis  can  be  a  relatively 
resource-intensive  effort,  and  the  research  team  has  developed  an  approach  to  streamline  the 
process  (i.e.,  “ROI  Lite”).  This  approach  involves  adoption  of  an  Early  Warning  System  that 
focuses  on  more  frequent  assessment  of  the  “vital  few”  leading  indicators  of  success/failure. 
Assessments  take  the  form  of  variance  analyses  for  key  ROI  variables  (e.g.,  acquisition  costs) 
and  less  frequent  re-visiting  of  the  overall  ROI  analysis  itself  (only  required  when  variances  are 
significant  and  suggest  that  either  performance  needs  to  be  improved  or  re-baselining  is 
necessary). 
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Integrating  Multiple  Perspectives 

Another  challenge  in  effectively  managing  SOA  performance  is  the  multitude  of 
conflicting  viewpoints  regarding  key  SOA  outcomes  and  which  particular  SOA-driven  benefits 
can  realistically  be  pursued.  This  is  often  the  result  of  confusion  in  benefit-related  terminology 
(e.g.,  “flexibility”  and  “time  savings”)  and  of  differing  stakeholder  needs.  For  instance,  a  primary 
benefit  expected  from  SOA  is  the  ability  to  expose  services  for  potential  re-use  by  other 
Government  entities,  which  is  typically  an  enterprise  viewpoint;  however,  an  executing  program 
viewpoint  could  realistically  be  that  the  expected  benefit  from  SOA  relates  to  garnering  flexibility 
to  quickly  respond  to  a  change  in  the  environment.  Since  SOA  supports  the  exposure  of 
services  with  the  intent  of  reuse,  challenges  also  include  the  need  to  manage  multiple  inter- 
Governmental  and  public-private  performance  relationships  and  uncertainty  associated  with 
future  service  demand  and  performance  requirements. 

Establishing  Stakeholder  Targets 

The  importance  of  addressing  multiple  SOA  viewpoints,  numerous  stakeholder  needs, 
and  the  uncertainty  associated  with  the  nature  and  magnitude  of  future  service  demand  each 
increase  the  complexity  associated  with  acquiring  necessary  services  and  capability  from  other 
Government  entities  and  commercial  industry.  Methods  that  address  these  limitations, 
challenges,  and  pressures  for  more  effective  and  efficient  SOA  lifecycle  performance 
management  have  not  been  widely  adopted  within  Government  settings.  Such  methods  are 
fundamental  to  determining  whether  both  SOA  business  (e.g.,  cost  savings  through  reuse)  and 
technical  (e.g.,  flexibility  to  meet  operational  needs)  targets  are  being  met  in  a  mission-needs 
context  and  to  manage  a  more  complex  stakeholder,  provider  and  consumer  environment. 

3.0  Applying  Service-level  Agreements  (SLAs)  to  Manage  SOA 
Service  Provisioning 


Service-level  Agreements  (SLAs)  have  been  a  highly 
recommended  and  time-tested  way  (in  some  environments)  to 
establish  performance-related  agreement  between  service  providers 
and  consumers  (other  methods,  such  as  Memoranda  of  Agreement, 
are  typically  applied  for  service  provider  relationships  between 
Government  entities)  (GSA,  DoD,  NASA,  2005).  And,  effective 
application  of  SLAs  can  help  address  some  of  the  challenges 
identified  in  Section  2.0  (e.g.,  expectation  management,  integrating 
multiple  perspectives,  and  measuring  non-fiscal  returns).  An  SLA  is 
a  formal,  negotiated  agreement  between  two  parties.  It  is  a  contract 
between  customers  and  their  service  providers,  and  it  records  the 
common  understanding  about  service  features  such  as  priorities,  responsibilities,  and 
guarantees. 

The  main  purpose  of  the  SLA  is  to  articulate  agreements  reached  on  the  level  of  service 
to  be  provided.  For  example,  it  may  specify  levels  of  availability,  serviceability,  performance, 
operation,  or  other  service  attributes,  such  as  billing  and  even  penalties  in  the  case  of  violation 
of  the  SLA  (“Service  level  agreement,”  2007).  SLAs  have  been  applied  for  almost  two  decades 
by  fixed-line  telecom  operators  as  part  of  their  contracts  with  corporate  customers.  More 
recently,  some  Information  Technology  (IT)  enterprises  have  adopted  the  idea  of  using  SLAs 
with  their  customers  to  allow  for  comparing  delivered  versus  promised  quality  of  service  (2007). 


Federal  Government 
Performance  Management 


SOA  Performance 
Management 


SOA  SIA  Governance 
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Application  of  Service-level  Agreements  (SLA)  is  a  recommended,  but  not  required, 
method  to  describe  performance  expectations  for  services  that  are  acquired  by  the  Government 
from  an  external  (i.e.,  commercial  industry)  service  provider,  and  the  use  of  SLAs  is  prevalent 
when  services  are  acquired  using  Performance-based  Acquisition  (PBA)  techniques. 

3.1  Government  Experiences  in  Applying  SLAs 

Government  agency  experiences  with  applying  SLAs  for  managing  contract  performance 
objectives  have  been  mixed.  In  some  instances,  when  SLAs  have  been  applied  and 
performance  objectives  are  not  effectively  achieved,  the  primary  reason  for  failure  is  that  the 
SLAs  that  were  initially  created  were  not  consistently  applied,  maintained,  and  updated  (as 
necessary)  throughout  the  contract  period  of  performance.  In  other  instances,  SLAs  fail  to 
support  effective  performance  management  because  they  are  managed  individually  and  without 
sufficient  consideration  of  how  all  SLAs  supporting  a  particular  contract  relate  to  one  another  to 
achieve  overall  outcomes. 

SLAs  are  often  exclusively  applied  as  a  transactional  and  computer-generated 
communication  of  performance  status,  which  minimizes  their  inherent  power  to  form  binding 
agreements  between  parties  who  may  have  competing  agendas.  When  efforts  are  undertaken 
by  the  Government  to  leverage  SLAs  as  a  means  of  achieving  and  maintaining  meeting-of-the- 
minds  between  a  service  provider  and  consumer,  they  are  often  difficult  to  enforce  because  of 
how  and  when  the  SLAs  were  connected  to  contractually  oriented  provisioning  agreements. 

Administration  of  SLAs  often  becomes  overly  resource-intensive,  and  Government 
agencies  are  sometimes  motivated  to  simply  replace  SLA  monitoring  efforts  with  other, 
potentially  less  authoritative,  monitoring  approaches.  Alternatively,  Government  organizations 
can  simply  become  so  involved  in  SLA  administration  that  they  understandably  lose  sight  of 
performance  interdependencies  and  exactly  what  performance  really  should  be  measured  to 
achieve  desired  outcomes. 

While  challenges  associated  with  effective  and  efficient  Government  SLA  application 
have  existed  for  many  years,  the  advent  of  SOA  and  increased  pressures  for  agile  service 
provisioning  in  web-enabled  environments  has  added  new  and  more  pervasive  challenges.  In 
these  service-oriented  environments,  managing  delivery  against  desired  outcomes  is  complex 
and  multi-dimensional,  e.g.,  SLAs  may  be  nested  and  may  be  dependent  on  separate 
application,  hosting,  and  communications/networks  performance  needs.  The  nature  and 
magnitude  of  future  service  demand  is  frequently  unclear.  And,  capabilities  will  likely  be  jointly 
created  and  maintained  by  numerous  internal  and  external  organizations.  The  Federal 
Acquisition  Regulation  {FAR)  and  other  government  procurement  policies  can  likely 
accommodate  service-oriented  provisioning  needs,  but  comprehensive  guidance  is  not  available 
to  support  Government  organizations  in  establishing  SLA  monitoring  systems  that  effectively 
address  performance  interdependencies  among  multiple  contributors  in  achieving  overarching 
capabilities. 
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3.2  Increasing  SLA  Effectiveness 

To  increase  the  effectiveness  of  SLAs,  they  should  state  in  measurable  terms: 

■  The  service  to  be  performed  and  outcome  expectations, 

■  Key  Performance  Indicators  (KPIs)  and  the  level  of  service  that  is  acceptable  for 
each, 

■  The  manner  by  which  service  is  to  be  measured  and  how  “success”  is  defined, 

■  The  parties  involved  and  the  responsibilities  of  each, 

■  The  reporting  guidelines  and  requirements,  and 

■  Incentives  for  the  service  provider  to  meet  the  agreed-upon  target  levels  of  quality. 
Figure  3-2  illustrates  recommended  relational  SLA  elements. 


SLA  Element 

Description 

2.1  CONTEXT 

Purpose/Background 

Description  of  what  the  SLA  has  been  designed  to  accomplish 

Stakeholders 

Identifies  who  cares  about  this  performance  and  what  they  care  most  about 

Service  Interdependencies 

Explains  how  the  SLA  and  work  scope  fit  into  the  entire  supply  chain 

2.2  SCOPE  OVERVIEW 

Business  Scope  and  Objectives  ||A  high  level  description  of  the  SLA’s  business  objectives 

2.3  SERVICE  DESCRIPTIONS 

Service  Descriptions  | 

(Detailed  description  of  the  services  being  provided  through  the  agreement 

2.4  KEY  PERFORMANCE  INDICATORS 

Service  Levels/Performance 
Metrics 

Required  performance  and  how  service  is  to  be  delivered 

Data  Requirements 

Data  to  be  provided  by  the  contractor  to  enable  performance  monitoring 

Security  Management 

Security  issues  relevant  to  services  provided 

Workload  Constraints 

Highest  expected  level  of  service  demand.  Degradation  schedule  if  excessive 
demand 

Severity  and  Priority  Levels 

Severity  levels  for  service  interruption/degradation;  service  restoration 
priorities 

2.5  ROLES  AND  RESPONSIBILITIES 

Roles  and  Responsibilities 

Mutually  agreed  upon  roles  along  with  corresponding  responsibilities  for  each 
team  member 

2.6  RECOURSE/REWARD  SCHEME 

Excused  Performance 

Conditions  under  which  the  contractor  will  not  held  to  the  Absolute  KPIs 

Escalation  Procedures 

What  actions  to  take  if  service  delivery  is  not  satisfactory 

Service  Level  Bonuses/Penalties 

Consequences  for  failing  to  meet  Absolute  KPIs;  rewards  for  superlative 
performance 

2.7  REPORTING  GUIDELINES  AND  REQUIREMENTS 

Required  Performance  Reports 

Vendor’s  performance  reports  to  be  delivered  to  government 

Update  Procedures 

How,  how  often,  and  by  whom,  SLA  should  be  updated 

Issues  Management  Procedures 

Responsibilities  for  surfacing  and  resolving  problems/issues 

2.8  GLOSSARY 

Glossary  of  Terms  | 

[Written  to  minimize  misinterpretations 

Figure  3-2.  Recommended  Relational  SLA  Elements 

Key  SLA  lessons  learned  that  should  be  considered  include: 
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■  Agree  to  existing  service  levels:  Some  Government  agencies  agree  that  the 
required  service  levels  will  be  set  at  existing  performance.  Doing  so  preserves  the 
current  service  that  the  new  contract  was  designed  to  improve  (Delaney,  2004). 

■  Agree  to  agree  on  service  levels:  Some  Government  agencies  agree  to  work  out 
service  levels  after  contract  award  (2004).  However,  once  the  contract  is  signed,  the 
deal  team  often  breaks  up,  and  the  provider  may  not  be  incentivized  to  subsequently 
agree  to  challenging  service  levels. 

■  Agree  to  fix  service  levels  at  initial  provider  performance:  Some  Government 
agencies,  with  no  basis  for  setting  service  levels,  agree  to  set  them  at  whatever 
levels  the  provider  can  achieve  during  the  initial  months  of  the  contract.  This  can 
give  the  provider  an  incentive  to  hold  down  service  levels  during  those  initial 
transitional  months,  that  is,  during  a  potentially  volatile  time  in  the  contract  term 
(2004). 

■  Set  the  appropriate  incentives:  Some  Government  agencies  overlook  the  idea  that 
the  provider  will  "manage  to  the  money."  For  example,  in  a  call  center  contract, 
agencies  might  set  a  service  level  of  "answer  90%  of  calls  within  two  minutes" 
without  realizing  that  they  are,  in  effect,  telling  the  provider  to  ignore  any  call  that's 
gone  over  two  minutes  in  favor  of  one  that  could  still  be  answered  in  two  minutes 
(2004). 

■  Don’t  ask  for  the  moon:  Government  agencies  should  be  careful  about  requiring 
unnecessarily  high  performance  commitments.  Providing  better  service  may  require 
the  provider  to  use,  for  example,  redundant  systems,  excess  capacity  and  better 
technology  (2004). 

■  Realize  less  is  more:  Government  agencies  should  make  SLAs  simple  and  familiar. 

■  Make  SLAs  measurable  and  actionable:  Agencies  should  only  collect  data  upon 
which  they  are  going  to  base  decisions;  they  should  then  pre-set  the  actions  that  will 
be  followed  if  metrics  do  not  hit  targets. 

■  Detail  the  unusual  areas  and  boiler  plate  the  rest:  “Must-haves”  should  be 
articulated  in  the  contract  itself. 

■  Describe  methods  for  withholding/reducing  fee:  Loss  of  business/productivity  is 
rarely  compensated  directly  by  a  service  provider.  Typically,  a  rebate  proportional  to 
the  shortfall  of  the  service  vs.  the  payment  is  provided  by  the  service  provider  in 
future  performance  evaluation  periods.  SLAs  typically  include  escalation  procedures 
and  conditions  under  which  the  provider  will  not  be  held  responsible  for  service 
failings. 

■  Incorporate  contract  language  that  allows  SLAs  to  be  changed:  This  language 
should  tie  to  milestones  as  SLA  changes  may  impact  cost/schedule. 

Key  reasons  for  failure  of  SLAs  include:  (a)  The  Government  lacks  well-defined 
requirements  at  the  time  of  Request  for  Proposal  (RFP)  issuance,  and  (b)  When 
Government/contractor  performance  interdependencies  exist,  the  Government  must  have 
enough  solid  data  on  its  own  performance  to  counter  contractor  challenges. 

3.3  SLA  Considerations  for  SOA  Environments 

Ideally,  IT  and  business  stakeholders  must  work  together  to  define  realistic  service-level 
criteria  for  SOA,  especially  for  web  services  (Wainewright,  2003).  While  traditional 
infrastructure  SLAs  typically  measure  “feeds  and  speeds,”  SOA  SLAs  will  often  need  to 
measure  completed  events.  Blending  IT  and  business  factors  will  require  dialogue  and 
feedback,  which  can  be  used  to  inform  the  performance  measurement  and  management 
processes.  While  the  notion  of  measuring  up  to  specific  technical  performance  benchmarks  is 
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well-established  in  the  IT  industry,  the  idea  of  defining  service-level  objectives  in  terms  of 
business  factors  is  less  familiar.  Preparing  and  executing  an  SLA  in  a  SOA  environment 
presents  special  challenges.  Government  organizations  should  follow  some  basic  steps  when 
they  craft  and  manage  SLAs  in  an  SOA  environment  to  mitigate  risks  associated  with 
complexity  and  uncertainty  (Perera,  2008). 

■  Define  desired  outcomes:  SLAs  can  support  an  articulation  of  desired  outcomes 
between  business  and  technology  sides  of  the  organization  (2008).  And,  it  is 
recommended  that  SLAs  align  with  the  overall  Concept  of  Operations  (CONOPS). 

By  design,  an  effective  CONOPS  will  define  the  operational  concept  relative  to 
overall  objectives  and  will  support  an  understanding  of  key  interdependencies.  In  a 
SOA  environment,  people  should  consider,  from  the  outset,  alternatives  to  business 
as  usual.  Certainly,  people  can  use  SOA  to  perform  the  same  business  tasks  that 
previous  software  performed,  but  this  perpetuation  can  ignore  the  opportunities  that 
SOA  is  supposed  to  create  for  flexibility  and  adaptability  to  changing  business  needs. 

■  Match  technical  requirements  to  business  needs:  Software  designers  must  select 
performance  indicators  for  technical  services,  including  service  availability, 
bandwidth  and  response  times.  Forrester  Research  uses  the  analogy  of  a  consumer 
using  an  automated  teller  machine  to  explain  how  technical  SLAs  should  be  crafted. 
“It’s  not  enough  that  you  put  your  card  and  Personal  Identification  Number  (PIN)  [in 
the  machine]  and  request  to  withdraw  cash.  There’s  an  expectation  of  how  fast  that 
will  happen,  the  level  of  reliability  and  the  level  of  security”  (Perera,  2008). 

Varied  business  needs  require  different  technical  thresholds.  A  military  targeting 
application  requires  the  highest  levels  of  availability,  whereas  a  civilian  data  analysis  tool  can 
probably  operate  at  degraded  performance  levels  outside  of  normal  working  hours. 

Because  SOA  applications  have  many  loosely  coupled  services,  SLAs  can  get 
complicated.  For  example,  software  designers  need  performance  guarantees  if  they’re  going  to 
reuse  a  service.  In  that  case,  a  technical  SLA  between  the  service  provider  and  the  service 
consumer  will  be  necessary.  Each  individual  service  might  be  in  compliance  with  its  technical 
SLA,  and  yet  the  overall  application  could  still  fail  to  meet  its  performance  benchmarks.  SLAs 
cannot  be  an  afterthought;  they  should  become  part  of  the  system  engineering  process  that 
occurs  when  SOA  application  developers  are  selecting  services  to  incorporate  or  reuse. 
“However,  from  a  user  standpoint,  a  SOA  application  should  have  one  SLA”  (2008). 

■  Monitor  performance:  A  technical  SLA  provides  information  as  to  what 
performance  is  expected  from  a  SOA  application,  but  how  does  one  know  if  the 
application  meets  that  benchmark?  DISA’s  Net-Centric  Enterprise  Services  (NCES) 
program  created  a  SOA  framework,  a  structured  method  for  monitoring  all  service 
information  going  back  and  forth.  According  to  Computer  Sciences  Corporation 
(CSC),  “The  common  framework  captures  [...]  service  information  regardless  of  the 
program  or  organizational  entity”  (Perera,  2008).  “Performance  monitoring  is  an 
essential  step  in  avoiding  pass-the-buck  arguments  about  who  is  responsible  for 
performance  failures.  Consider  a  scenario  in  which  a  service  provider  agrees  to 
accept  10,000  consumer  data  queries  in  an  hour.  The  consumer’s  service 
information  shows  that  the  queries  are  not  exceeding  that  level,  but  the  application 
isn’t  responding.  Logs  show  that  the  consumer  sent  batches  of  10,000  requests  in 
ten  seconds  in  hourly  pulses,  and  batch  processing  wasn’t  part  of  the  original  SLA” 
(2008). 
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■  Enforce  the  agreement:  An  agreement  to  provide  service  without  a  mechanism  to 
penalize  noncompliance  is  not  much  of  an  agreement.  But,  this  can  sometimes 
occur  with  SOA  SLAs.  “A  user  agency  could  say  it  has  an  SLA  that  guarantees 
performance  levels,  but  a  provider  agency  could  argue  that  Congress  doesn’t  intend 
for  the  money  it  appropriates  to  the  provider  agency  to  be  used  to  fix  another 
agency’s  IT  problems”  (2008).  Although  under  various  laws,  notably  the  Economy 
Act  of  1933,  agencies  can  contract  for  services  from  another  agency,  the  law  when 
applied  to  SOA  “gets  into  some  sticky  areas  that  are  way  out  of  the  purview  of  IT 
people,”  said  Randy  Hite,  director  of  IT  architecture  and  systems  issues  at  the 
Government  Accountability  Office  (GAO).  “It  starts  getting  lawyers  involved.”  Partly 
because  of  those  legal  and  funding  issues,  SOA  studies  show  that  only  5%  of 
reusable  services  actually  are  reused  (2008).  It’s  easy  to  find  examples  of 
organizations  failing  to  fulfill  their  SLA  agreements.  For  that  reason,  SLAs  in  the 
federal  government  are  most  effective  within  a  single  organization  whose  various 
parts  are  supported  by  the  same  source  of  funding.  Not  going  outside  the 
organization  for  reusable  services  is  perceived  as  prudent.  That  constraint  doesn’t 
necessarily  apply  to  contracting  with  vendors  for  SOA  services.  Government 
agencies  can  try  to  financially  penalize  a  vendor  for  reusable  services  that  fall  short 
of  agency  expectations.  However,  vendors  are  not  eager  to  assume  extra 
responsibility  without  getting  paid  (2008). 

4.0  A  SOA  SLA  Governance  Framework 


Government  agencies  should  consider  adopting  an  SLA 
governance  framework  to  ensure  that  SLAs  can  be  as  effective  as 
possible  in  managing  performance  and  achieving  overall  outcomes. 
Such  a  framework  can  help  rationally  manage  all  the  individual 
performance  agreements  and  monitoring  activities,  especially  when 
the  Government  is  contracting  for  multiple  and/or  complex  services. 
SLA  governance  is  the  ongoing  process  of  reviewing  performance 
measures  and  contrasting  those  results  to  the  stated  goals  and 
targets.  Objectives  of  an  SLA  governance  framework  are  to  ensure 
that: 


■  Performance  standards,  as  communicated  through  SLAs,  provide  a  clear 
understanding  of  how  well  the  contractor  is  achieving  overall  service  contract  goals; 

■  SLAs  continue  to  describe  performance  deemed  critical  at  the  moment  to 
achievement  of  overall  outcomes; 

■  SLAs  and  performance  measures  are  prioritized  according  to  their  importance  in 
achieving  overall  outcomes;  and 

■  All  activities  and  surveillance  are  undertaken  as  effectively  as  possible  in  order  to 
assess  how  effectively  the  provided  services  support  the  overall  desired  outcomes. 


Figure  4-1  illustrates  the  purpose,  goals,  and  key  success  drivers  of  an  SLA  governance 
framework. 
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Figure  4-1.  SLA  Governance  Framework  Purpose,  Goals,  and  Key  Success  Drivers 

An  SLA  governance  framework  should  be  designed  so  that  all  SLAs  currently  being 
applied  to  monitor  performance: 

■  Meaningfully  describe  progress  toward  achievement  of  specific  outcomes  in  the 
context  of  overall  contract  objectives  and  in  consideration  of  SLA  interdependencies 
that  may  exist;  and 

■  Are  objectively  measured  at  the  appropriate  times  and  continually  serve  as  the 
primary  mechanism  for  objectively  determining  service  provider  payment  and 
incentives  (both  positive  and  negative). 

The  framework  should  assist  Government  leaders,  contracting  personnel,  and  Program 
Managers  in  consolidating,  synthesizing,  and  rationalizing  information  related  to  service 
performance  on  a  continuous  basis — in  such  a  way  that  performance  status  can  be  accurately 
determined  at  any  point  in  time  and  readily  translated  into  a  robust  characterization  of  how 
effectively  vital  outcomes  are  being  achieved.  The  SLA  governance  framework  should  clearly 
identify  key  service  provisioning  stakeholders,  their  performance  expectations,  and  if/how  their 
performance  expectations  are  being  satisfied.  The  framework  should  enable  the  maintenance 
and  improvement  of  service  quality  through  a  continuous  cycle  of  agreeing,  monitoring,  and 
reporting  upon  service  achievements  and  instigation  of  actions  to  eradicate  poor  service.  To  be 
effective,  the  SLA  governance  framework  should  define  roles  and  responsibilities  for 
performance  measurement  and  management.  The  framework  should  also  define  the  types  of 
performance  reviews  that  need  to  be  conducted  and  the  timing  of  these  reviews. 

SLA  governance  does  not  begin  when  the  SLA  itself  is  documented;  rather,  governance 
refers  to  managing  the  entire  process  throughout  the  acquisition  lifecycle.  The  initial  evaluation 
of  current  practices  before  the  document  is  started,  the  writing  of  the  SLA,  the  determination  of 
key  SLA  participants  and  associated  roles/responsibilities,  the  monitoring  of  the  effort’s 
progress,  as  well  as  the  need  for  any  changes  or  updates  to  the  agreement  are  all  part  of  the 
governance  process.  This  framework  provides  guidance  for  those  leaders  within  the 
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Government  who  are  responsible  for  overseeing  this  process  to  ensure  that  the  goals  laid  out  in 
the  SLA  are  realized.  The  government  and  service  providers  must  manage  the  relationship  on 
an  on-going  basis  by  continuously  monitoring  performance,  changing  business  needs  and 
updating  benchmarks.  At  regularly  agreed-upon  intervals,  the  government  should  determine 
whether  existing  contracts  need  to  be  modified  and  new  SLAs  drafted.  Figure  4-2  details  the 
proposed  governance  framework  and  the  steps  included  in  each  of  the  four  stages — Prepare, 
Create,  Monitor  and  Update. 


SLA  Governance  Framework 


PREPARE 


CREATE 


MONITOR 


Review  Current 
Business  Practices 


Establish  Goals 
of  Effort 


Define  how  SLA 
fits  in  with  goals 


ID  any 

Interdependencies 
with  other  SLAs 


Identify  Relevant  & 
Measurable  KPI’s 

Set  key  roles 
&  assign 
responsibilities 

Establish  Reporting 
Guidelines 


UPDATE 


All  metrics 

continually  measured 
&  monitored 

Key  roles  are  being 
held  accountable  for 
responsibilities 

Overall  effort  is 
moving  forward: 
•On  schedule 
•Benefits  being 
realized 

•Collaboration  by 
all  parties 


Figure  4-2.  Proposed  SLA  Governance  Framework 

Essential  steps  to  successful  SLA  management  include: 

■  Define  a  service  in  understandable  language.  This  is  the  service.  This  is  what 
it  means.  This  is  what  is  supported  and  what  is  not  supported.  This  is  how  it 
will  be  reported,  communicated,  charged. 

■  Understand  the  costs  at  a  granular  level,  identifying  all  the  different  cost 

elements  involved  in  the  delivery  of  a  service.  This  will  give  IT  the  ability  to 
also  execute  improvement  programs  aimed  at  further  reducing  these  costs. 

■  Price  the  service  delivery  accordingly.  There  will  be  projects  in  the  future  for 
which  the  business  may  not  immediately  see  the  value.  So  price  some  of  the 
services  to  allow  for  some  buffer  to  pay  for  these  yet-to-be-accepted  services. 

■  Implement  differentiated  charge-backs  to  reflect  the  differentiated  levels  of 

service  you  have  on  offer. 

■  Have  regular  service  reviews.  Reviews  are  a  communication  and  marketing 
mechanism  for  IT  to  show  to  business  how  it  is  improving  and  helping  the 
business.  Identify  what  else  is  needed  by  the  business  through  this  dialogue. 
A  feedback  loop  is  thus  created  in  which  both  business  and  IT  are  able  to 
help  each  other  improve  (“Managing,”  2007). 
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5.0  Conclusions 


Mechanisms  are  currently  not  consistently  in  place  within  the  Federal  Government  for 
programs  to  identify  key  stakeholders,  quantitatively  articulate  stakeholder  needs,  and 
quantifiably  assess,  on  a  timely  basis,  whether  stakeholder  needs  have  been  satisfied.  The 
Comptroller  General  asks  federal  programs  and  agencies  to  improve  performance  management 
by: 


■  Comprehensively  reassessing  what  the  federal  government  does  and  how  it  does  it; 
reconsidering  whether  to  terminate  or  revise  outdated  programs  or  services 
provided. 

■  Reexamining  the  beneficiaries  of  federal  programs;  reconsidering  who  is  eligible  for, 
pays  for,  and/or  benefits  from  a  particular  program  to  maximize  federal  investments. 

■  Improving  economy,  efficiency,  and  effectiveness  of  federal  operations;  capturing 
opportunities  to  reduce  costs  through  restructuring  and  streamlining  federal  activities. 

■  Attacking  activities  at  risk  of  fraud,  waste,  abuse,  and  mismanagement;  focusing  on 
minimizing  risks  and  costs.  (Mihm,  2000,  July  20) 

MITRE  research  on  performance  management  has  resulted  in  recommendations  to:  (1) 
apply  ROI  analysis  principles  as  the  foundation  for  more  effective  performance  management  of 
Government  SOA,  (2)  create  comprehensive  SLAs  to  articulate  agreements  between  the 
Government  and  external  service  providers,  and  (3)  manage  SLAs  through  a  governance 
framework. 

SOA  involves  multiple  and  complex  participants  (e.g.,  SOA  developers,  service 
providers,  service  consumers)  and  organizations  (e.g.,  multiple  Government  organizations  and 
commercial  industry).  It  also  involves  potential  uncertainty  associated  with  future  performance 
expectations  as  services  are  exposed  through  the  SOA;  the  nature  and  magnitude  of  future 
demand  for  services  will  likely  not  be  known  with  certainty  at  the  outset.  Careful  planning  must 
be  undertaken  by  Government  organizations  to  determine  outcomes  for  multiple  stakeholders 
and  determine  how  those  outcomes  are  translated  to  performance  expectations  that  will  be 
communicated  to  service  providers. 

If  SLAs  are  applied  to  support  on-going  SOA  performance  management,  then  efforts 
should  be  undertaken  to  directly  connect  these  SLAs  with  technical  performance  requirements 
and  ultimate  SOA  expectations.  The  SLAs  should  be  carefully  crafted  to  ensure  that  flexibility 
for  the  Government  to  evolve  performance  expectations  is  maximized. 

For  SLAs  to  be  effective,  a  disciplined  governance  process  must  be  undertaken  by 
sponsors  to  ensure  that  the  SLAs  are  actually  measured  and  monitored.  On  a  timely  basis,  the 
SLAs  should  be  re-evaluated  to  determine  whether  they  are  actually  measuring  something  of 
importance  and  are  still  relevant  to  outcomes. 

The  problem  with  SLAs  is  that  once  the  ink  has  dried,  the  provision,  monitoring,  and 

management  of  these  agreements  can  become  the  bone  of  contention  between  the 

people  who  are  left  to  execute,  monitor  and  manage  the  contract.  The  need  to  manage 

SLAs  is  becoming  a  necessity  if  SLAs  are  to  achieve  any  semblance  of  success. 

Without  management,  SLAs  are  like  cars  that  go  wildly  off  a  highway.  You  need  checks 
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and  balances  to  make  sure  that  all  concerned  are  running  in  the  same  direction  and 
hopefully  meeting  all  the  obligations  set  forth  in  the  contract.  (“Managing,”  2007) 


With  a  performance  management  program  in  place,  well-written  and  governed  SLAs 
support  government  programs  and  provide  transparent  accountability  to  their  stakeholders. 
Transparent  accountability  can  support  the  Government  in  addressing  challenges  associated 
with  complexity  and  uncertainty. 
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Appendix  A.  Acronyms 


CSC 

Computer  Sciences  Corporation 

CONOPS 

Concept  of  Operations 

DISA 

Defense  Information  Systems  Agency 

FAR 

Federal  Acquisition  Regulation 

FCW 

Federal  Computer  Week 

GAO 

Government  Accountability  Office 

IT 

Information  Technology 

KPI 

Key  Performance  Indicator 

MOIE 

Mission-oriented  Investigation  and  Experimentation 

NCES 

Net-centric  Enterprise  Services 

PBA 

Performance-based  Acquisition 

PIN 

Personal  Identification  Number 

RFP 

Request  for  Proposal 

ROI 

Return  on  Investment 

SLA 

Service-level  Agreement 

SOA 

Service-oriented  Architecture 
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2003  -  2009  Sponsored  Research  Topics 

Acquisition  Management 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  BCA:  Contractor  vs.  Organic  Growth 

■  Defense  Industry  Consolidation 

■  EU-US  Defense  Industrial  Relationships 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Managing  Services  Supply  Chain 

■  MOSA  Contracting  Implications 

■  Portfolio  Optimization  via  KVA  +  RO 

■  Private  Military  Sector 

■  Software  Requirements  for  OA 

■  Spiral  Development 

■  Strategy  for  Defense  Acquisition  Research 

■  The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 
Contract  Management 

■  Commodity  Sourcing  Strategies 

■  Contracting  Government  Procurement  Functions 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 

■  Navy  Contract  Writing  Guide 

■  Past  Performance  in  Source  Selection 

■  Strategic  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  USAF  Energy  Savings  Performance  Contracts 

■  USAF  IT  Commodity  Council 

■  USMC  Contingency  Contracting 
Financial  Management 

■  Acquisitions  via  leasing:  MPS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities-based  Planning 

■  Capital  Budgeting  for  DoD 
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■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition  Budgeting 
Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 
Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-tern  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 
Logistics  Management 

■  Analysis  of  LAV  Depot  Maintenance 

■  Army  LOG  MOD 

■  ASDS  Product  Support  Analysis 

■  Cold-chain  Logistics 

■  Contractors  Supporting  Military  Operations 

■  Diffusion/Variability  on  Vendor  Performance  Evaluation 

■  Evolutionary  Acquisition 

■  Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 

■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance  Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

■  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  Aegis  Microwave  Power  Tubes 
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Sense-and-Respond  Logistics  Network 
Strategic  Sourcing 


Program  Management 


Building  Collaborative  Capacity 

Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 
Collaborative  IT  Tools  Leveraging  Competence 
Contractor  vs.  Organic  Support 

Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

KVA  Applied  to  Aegis  and  SSDS 

Managing  the  Service  Supply  Chain 

Measuring  Uncertainty  in  Earned  Value 

Organizational  Modeling  and  Simulation 

Public-Private  Partnership 

Terminating  Your  Own  Program 

Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 


A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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Purpose  and  Context 


□  Purpose 


Describe  application  of  Return-on-lnvestment  (ROI)  analysis  for  SOA 
performance  management 


Discuss  use  of  SLAs  to  articulate  agreements  between  the  Government  and 
external  service  providers  in  SOA  environments 


Recommend  an  SLA  Management  governance  framework 


□  Context 

Summarizes  results-to-date  of  MITRE  Corporation  (MITRE)  research  efforts: 

-  Stakeholder-Driven  Performance  Management 
n  K.  Buck,  L.  Oakley-Bogdewic 

SOA  Performance  Measures  Expression  in  Performance-Based  Acquisition  (PBA) 
Vehicles 
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Performance  Management  -  Federal  Focus 


□  Objectives  must  be  outcome-driven 

Complying  with  performance  reporting  is  insufficient 
Derive  outcomes  from  stakeholder-articulated  statements 


To  Address 
V-  Complexity  and 
Uncertainty 


Mission,  vision,  and  strategic/transition 


□  Use  a  shared  responsibility  model 

Program/portfolio/contract  managers  all  have  vested  interest 

□  Federal  government  has  a  lackluster  performance  management  track  record 

Clearest  evidence:  failed  programs  (failure  to  deliver  or  significant  over-budget/over- 
schedule  delivery) 

□  Performance  management  is  challenging  for  the  federal  government 

Challenges  became  greater  with  dependencies  on  non-controlled  resources,  e.g.,  networks 
Service-Oriented  Architecture  (SOA)  significantly  increases  dependencies 

Uncertainty  and/or  complexity  increase 

Data  and  functions  now  added  to  the  non-controlled  resource  pool 
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Performance  Management -Federal  Focus 


□  Key  hypotheses 

if 


Then 


So 


Complexity 


t 


#  of  moving 
parts  and 
relationships 


t 


Focus  more  on 
interdependencies 


Uncertainty 
about  the 
future 


t 


Chances  of 
reducing 
uncertainty  over 
time 


t 


Plan  for  contingencies 
and  garner  flexibility 


%  of  DoD  IT 
spending 
contracted  out 


=  79.3% 


[V 


Contract 
performance 
impact  on  IT 
performance 


=  significant 


Manage  IT  delivery  as  a 
tight  linkage  between 
contract  and  overall 
performance 
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SOA  Performance  Management 

□  SOA  definition 

-  An  architectural  approach  used  to  build  solutions 

Consists  of  a  set  of  services,  service  consumers,  service 
producers  and  mechanisms  for  discovery  and  establishing 
contracts 


Federal  Government 
Performance  Managemen 


To  Address 
V-  Complexity  and 
Uncertainty 


□  SOA  helps  organizations  share  resources,  and  it 
increases: 


SOA  Construct 


Acquisition  complexity:  multiple  relationships  to  manage 
Uncertainty  regarding  when  resource  needs  change 

□  SOA  performance  management  challenges: 


Expecting  savings  without  analysis 
Understanding  the  SOA  lifecycle 
Measuring  non-fiscal  returns 
Managing  expectations 
Effectively  managing  SOA  is  resource-intensive 
Integrating  multiple  perspectives 
Establishing  stakeholder  targets 


Many  more 
with 

Differing  needs 


SOAP 


SOAP  =  Simple  Object  Access  Protocol 
UDDI  =  Universal  Description ,  Discovery , 
and  Integration 

WSDL  =  Web  Services  Description 
Language 
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SOA  Performance  Management 

□  Without  credible  and  relevant  benchmarks  for  what  performance  can  be 
expected  from  your  SOA  participation,  how  will  you  assess  SOA  performance? 


SOA  Participant 


Mission 


Current  Capability 


Performance  Gap 


Desired 

Capability 

ap  _ I 


ROI  calculation  = 
what  we  learned  in  finance  class 

ROI  analysis  = 

ROI  calculation  +  assessment  of  intangibles 


SOA  investment 
option 


ROI  Analysis 


Flexibility 


Cost  Savings 


Time  Savings 


Management 

Efficiencies 


Agility 


Opportunity 
to  Innovate 


Projections 


Investment  costs  and  benefits 
can  be  easily  monetized? 

(e.g.,  salvage  value) 


No 


Investment  costs  and  benefits 
can  be  monetized,  but  not 
easily? 

(e.g.,  productivity) 


Yes 


No 


Investment  costs  and  benefits 
can  be  quantified,  but  not 
accurately  monetized? 

(e.g.,  customer  satisfaction) 


No 


Investment  impacts  cannot  be 
accurately  expressed 
monetarily? 

(e.g.,  degree  of  regulatory 
compliance) 


Yes 


Cost,  Economic, 
and/or  Financial 
.  Analysis 


Yes 


EXAMPLE  OECISION 
ANALYTIC  APPROACHES 


Project  Scorecard 


ROI  Calculation 
Suite  of  Metrics 


Net  Present 
Value 
(NPV) 


Internal 
Rate  of 
Return  (IRR) 


Borda  Voting 


MulthAHribute  Utility  Theory 

Real  Options  Theory 


Balanced  Scorecard 


Other 


V 


U  ncertai  nty  Assessment 


-  List  of  priorities 

-  List  of  relative  desirability 

-  Comparative  customer  satisfaction  ratings 

-  Balanced  Scorecard  ratings 

-  Number  of  votes  "for"  and  "against" 


Yes. 


Qualitative 

Assessment 


(1)  What  are  the  social  consequences? 

(2)  What  are  the  strategic  implications? 

(3)  What  is  the  effect  on  employee  morale? 

(4)  What  are  the  political  ramifications? 

(5)  Stoplight  matrix  of  risk  assessment 

y 


ROI-based 

investments 

selection 


Performance 

management 

planning 

(to  include 
defining  ROI 
based  metrics) 


Performance 

Monitoring 

>  (as  an  on-going 
comparison  of 
initially  expected 
versus  actual 
ROI) 


Post-Investment 

Review 
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t  Applying  SLAs  to  Manage  SOA  Service 

l-d  Comer  *  *  • 

Provisioning 


■=> 


□  SLAs  are  a  means  to  establish  performance-related  agreements  between 
service  providers  and  consumers 

□  An  SLA  is  a  formal  negotiated  agreement  between  two  parties 

-  A  "contract"  between  customers  and  their  service  providers 

-  Records  the  common  understanding  about  service  features 

□  Government  experiences  in  applying  SLAs: 

-  Results  have  been  mixed,  often  depending  on  whether  the  SLAs  were: 

n  Consistently  applied,  maintained  and  updated 

n  Managed  with  awareness  of  interdependent  SLAs'  performance  needs 

n  Computer  transaction-based  vice  an  accurate  vehicle  for  consumer-provider 
communication 

-  "Losing  the  forest  for  the  trees" 

n  SLA  administration  focus  vice  continuous  assessment  as  to  whether  desired  outcomes 


are  being  achieved 
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Applying  SLAs  to  Manage  SOA  Service 


Provisioning 

□  Transactional  versus  Relational  SLAs 

Transactional 

Measures  moment-in-time 
performance  indicators 
(e.g.,  throughput),  typically  in  an 
automated  fashion 

-  Relational 

Why  performance  levels  are  needed 

Parties7  agreement  about  performance 
standards 

Rationale  for  thresholds 
Recourse  for  performance  lapses 


Relational  SLA  Elements 

CONTEXT 

Purpose/Background 

Stakeholders  and  how  this  SLA  supports  their  needs 

Service  Interdependencies 

SCOPE  OVERVIEW 

Business  Scope  and  Objectives 

SERVICE  DESCRIPTIONS 

KEY  PERFORMANCE  INDICATORS 

Service  Levels/Performance  Metrics 

Data  Requirements 

Security  Management 

Workload  Constraints 

Severity  and  Priority  Levels 

ROLES  AND  RESPONSIBILITIES 

RECOURSE/REWARD  SCHEME 

Excused  Performance 

Escalation  Procedures 

Service  Level  Bonuses/Penalties 

REPORTING  GUIDELINES  AND  REQUIREMENTS 

Required  Performance  Reports 

Update  Procedures 

Issues  Management  Procedures 

GLOSSARY  OF  TERMS 
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Applying  SLAs  to  Manage  SOA  Service 
Provisioning 


□  SLAs  are  most  effective  if  stated  in  measurable  terms  and  include 

-  Stated  expectations  of  outcomes 

Derived  Key  Performance  Indicators  (KPIs)  with  outcome-supported  service  levels 

-  Defined  measurement  mechanisms  and  what  determines  "success" 

-  Parties  involved  and  their  responsibilities 

-  Reporting  guidelines  and  requirements 

-  Service  provider  incentives  for  meeting  agreed-upon  levels  of  quality 


□  Checklist  for  determining  if  SLAs  are  effectively  supporting 
objectives: 

0  SLAs  articulate  a  definitive  performance  provider-consumer 
agreement 

0SLAs  clearly  align  with  overall  enterprise  performance 
objectives 

Align  decision  frameworks,  documents,  and  contract 
artifacts 

■  Involves  diligent  configuration  control  and  communication 


0 


SLAs  allow  for  re-negotiation  if  future  conditions  significantly 
change 
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Applying  SLAs  to  Manage  SOA  Service 
Provisioning 


□  Key  reasons  for  SLA  failure: 

-  Lack  of  well-defined  requirements  at  the  time  of  RFP  issuance 

When  performance  interdependencies  exist,  each  party  must  have  solid  data 
on  its  own  performance  to  counter  challenges 


□  Key  SLA  lessons  learned  ^  : 

Agree  to  pre-existing  service  levels 


[1]  “The  Outsourcing  Revolution,  2004:  Protecting 
Critical  Business  Functions ”  J.  Delaney 


n  Some  Government  agencies  agree  that  the  required  service  levels  will  be  set 
at  pre-existing  performance  levels.  By  doing  so,  they  preserve  the  current 
service  that  the  new  contract  was  designed  to  improve 


Agree  to  service  levels  before  contract  award 

n  There  is  little  incentive  to  uphold  post-contract  award  agreements 


-  Do  not  agree  to  fix  service  levels  at  initial  provider  performance 

n  This  reduces  flexibility  and  incentive  to  get  better  performance  if  future  needs 
are  not  known 


-  State  incentives  appropriately 

n  Ensure  incentives  reinforce  positive  behavior  expectations 

-  Don’t  ask  for  the  moon 


MITRE 
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Applying  SLAs  to  Manage  SOA  Service 
Provisioning 


□  More  key  lessons  learned 

-  Less  is  more — make  SLAs  simple  and  familiar 

-  Make  SLAs  measurable  and  actionable 

n  Collect  data  to  be  acted  on 

n  Predetermine  actions  to  be  undertaken  when  metrics  fall  short 

-  Detail  the  unusual  areas  and  boiler  plate  the  rest 

n  “Must  haves”  should  be  articulated  in  the  contract  itself 

-  Describe  methods  for  withholding/reducing  fee 

n  Loss  of  business/productivity  is  rarely  compensated  directly 
-  Typically,  a  rebate  proportional  to  the  shortfall  is  used 

n  SLAs  typically  include  escalation  procedures  and  conditions  under  which 
they  are  invoked 

Incorporate  contract  language  that  allows  SLAs  to  be  changed 

n  This  language  should  tie  to  milestones  as  SLA  changes  may  impact 
cost/schedule 
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SOA  SLA  Governance 


□  SLA  governance  is  the  ongoing  process  of  reviewing  performance 
measures  against  stated  goals  and  targets  and  reassessing  SLA  value 


□  The  ultimate  objectives  of  an  SLA  governance  framework  are  to  ensure: 

Performance  standards  are  established  to  meet  overall  contract  goals 
SLAs  continue  to  describe  performance  deemed  critical  to  overall  outcomes 
SLAs  and  performance  measures  are  prioritized 
All  activities  and  surveillance  are  undertaken  as  effectively  as  possible 

□  What  we  are  trying  to  avoid  with  an  SLA  Governance  Framework? 


Per  the  Government  Accountability  Office  (GAO)  in  reviewing  Navy-Marine  Corps  Intranet 
(NMCI)  performance:  “The  Navy  defined  strategic  goals  for  its  NMCI  program  and 
developed  a  plan  for  measuring  and  reporting  on  achievement  of  these  goals.  However, 
the  Navy  did  not  implement  this  plan,  choosing  instead  to  focus  on  defining  and 
measuring  contractually  specified  SLAs.  Program  officials  did  not  have  performance 
data  to  demonstrate  progress  in  relation  to  strategic  goals.  Without  effective 
performance  management,  the  Navy  is  increasing  the  risk  that  the  program  will  continue 
to  fall  short  of  its  goals  and  expected  results.”  GAO-07-51,  December  2006,  pp.  18-19. 
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□  SOA  SLA  governance  goals  and  success  drivers 

$LA  Governance  Framework 


Purpose 


Outcomes  are 
achieved 
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SOA  SLA  Governance 


□  GAO  report  m  on  NMCI  effort  strongly  suggests  that 
downside  risk  associated  with  SLAs  is  “getting  lost  in  the 
trees  for  the  forest” 

[1]  GAO  Report  GAO-07-51.  “DOD  Needs  to  Ensure  That  Navy  Marine  Corps  Intranet  Program  Is  Meeting  Goals  and  Satisfying  Customers” 

Suggested  SLA  Governance  Framework  to  ensure  SLAs  are  effectively 
managed  and  align  with  expected  outcomes: 

SLA  Governance  Framework 


MITRE 


PREPARE 


Review  Current 
Business  Practices 


Establish  Goals 
of  Effort 


Define  how  SLA 
fits  in  with  goals 


ID  any 

Interdependencies 
with  other  SLAs 


CREATE 


* 


Identify  Relevant  & 
Measurable  KPI’s 

Set  key  roles 
&  assign 
responsibilities 

Establish  Reporting 
Guidelines 


UPDATE 


Necessary  updates 
identified  and  made 


* 

* 


MONITOR 


All  metrics 

continually  measured 
&  monitored 

Key  roles  are  being 
held  accountable  for 
responsibilities 

Overall  effort  is 
moving  forward: 
•On  schedule 
•Benefits  being 
realized 

•Collaboration  by 
all  parties 

Identify  areas  in 
SLA  which  require 
updates  or  changes 
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